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Abstract. In this article we study the symmetry breaking phenomenon of solutions of non- 
cooperative elliptic systems. We apply the degree for G-invariant strongly indefinite functionals 
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to obtain simultaneously a symmetry breaking and a global bifurcation phenomenon. 



Qh ! 1. Introduction 

In this paper, we consider a symmetry breaking of solutions of non-cooperative elliptic sys- 
tems of the form: 

{ -A Wl = V Wl F(w 1 ,w 2 ) +fx in Q 

Aw 2 = V W2 F(w 1 ,w 2 ) + f 2 in fi (1) 
§p_ _ dp _ q on 9n 

I I 1 

, where M n is an orthogonal representation of a compact Lie group G, fi C M. n is an open, 
bounded, G-invariant set with a smooth boundary and F e G 2 (IR 2 ,IR). That is we discuss the 
existence of a G-symmetric function (fi, f 2 ) such that there is a f^-symmetric solution (wx, w 2 ) 
of system p]), where K is a closed subgroup of G. If such a solution exists, we say that occurs 
CO ! a symmetry breaking of solutions of problem ([1]). 

The problem of symmetry breaking has been studied by many authors under various assump- 
tions on F and fi, see for instance [3]- [7], [121 Ellin], [H]-[22], [25]. Of course this list is far from 
being complete. The authors have used different tools to obtain their results: Rybakowski's 
homotopy index, the equivariant Conley index or the Leray-Schauder degree. We have applied 
the degree for G-invariant strongly indefinite functionals, see [T5] , to obtain our results. Using 
this degree we have formulated conditions on F which enable us to decide whether there is a 
connected set of solutions of the main problem. 

The idea of the proof of our main result is to reduce the problem to a bifurcation one. 
We follow the idea from [6], due to Dancer. The author has used a different tool, that is 
Rybakowski's homotopy index, see [23], which cannot be used to prove our results, because the 
functional corresponding to system ([[]) is strongly indefinite. Moreover, using Rybakowski's or 
Conley indices it is only possible to obtain a sequence of solutions of the symmetry breaking 
problem. Using the degree for G-invariant strongly indefinite functionals we have obtained a 
global bifurcation of solutions that problem. Moreover, our method can be used to handle a 
number of related problems. 
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After this introduction our article is organised as follows. 

In section 2 we introduce our notation and reduce the symmetry breaking problem to a 
bifurcation problem. 

In section 3 we consider a system of elliptic equations and recall basic properties of the 
operator induced by this system. We formulate the symmetry breaking and the corresponding 
bifurcation problem for this system. We calculate the degree for A(i^)-invariant strongly 
indefinite functionals for an operator associated with a linear system of equations, where N(K) 
is the normalizer of a subgroup K of G. We use this results to proceed some computations in 
a nonlinear case. 

In section 4 we formulate and prove the main results of this article. To do it we use the 
abstract results from the previous sections. 
In section 5 we illustrate our method. 

To make this article self-contained, we have included in section 6 the definition of the Euler 
ring U (G) of a compact Lie group G and the definition and basic properties of the degree for 
G- invariant strongly indefinite functionals, due to Gol§biewska and Rybicki, see [T3] . 

2. Preliminaries 

Throughout this article G stands for a compact Lie group and sub(G) for the set of closed 
subgroups of G. Let (H, (•, •)) be a separable Hilbert space, which is an orthogonal repre- 
sentation of G and let H K = {x G H : V g eK gx = x} be the set of all fixed points of the 
action of a subgroup K G sub(G). The set N{K) is the normalizer of a subgroup K G sub(G), 
i.e. N{K) = {g G G : gK = Kg}. Fix k G N. Let C£(H,R) denote the set of all G- 
invariant functionals of class C h , i. e. ^f(gx) = ^(x), where \& G Cq(H, R), g G G x G H, and 
C7^.~ 1 ('H,'H) the set of all G-equivariant operators of class C fc_1 , i. e. T(gx) = gT(x), where 
T G C k G ~ l (H, H), g G G, x G U. It can be easily shown that for a fixed K G sub(G), U G C H K 
and if * G C%(H,R), then the gradient W G C^^U), keN.We denote by S 7 (7£,p) the 
open unit ball in Ti centered at a point p of radius 7. Moreover, we put B(%,p) = Bx(%,p), 
-B 7 CH) = B 7 (T-L,0) and B(H) = Bi(H,Q). Suppose that A is a linear space of parameters, 
* G C%{U x A, R) is such that V^(0, A) = for every A G A. Consider the equation 

V tt *(w,A)=0. (2) 

Define a set of non-zero solutions of (j2J) by A/" = {(w, A) G ("H \ {0}) x A : V M ^(u, A) = 0}, fix 
A G A and denote by C(A ) a connected component of the closure cl(A/") such that (0, A ) G 
C(A ). 

Definition 2.1. A point (0, Ao) G {0} x A is said to be a local bifurcation point of solutions 
of equation (|2J), if (0, A ) G cl(A/"). A point (0, A ) G {0} x A is said to be a branching point of 
non-zero solutions of equation (j2J), if C(A ) 7^ {(0, Ao)}. A point (0, A ) G {0} x A is said to be a 
global bifurcation point of non-zero solutions of equation (j2J), if either C(Ao)n(Ox (A\{Ao}) 7^ 
or C(Ao) is not bounded. 

Problem 1. Let T G C%{U,U). Does there exist w G U K \U G such that T(w) G H G ? 
For subspaces K 2 C ^1 C H set Hi 9 M 2 = {ti G Hi : (w, f ) = Vt, e -^ 2 }. 
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Consider a G-equivariant projection tt: Ti — > M. such that ini7r = (H G ) ± . Then im(J — 71) = 
U G and note that tt(U k ) C H k for every K G sub(G). Define tt x : U k ->• U to be the 
composition 7i"i = 7roi, where i : K K — >■ {%Q'H K )®'H K is the embedding given by = (0, 2). 
The mapping z is Af(A^)-equivariant (the space l-i K is (A^-invariant and does not have to be 
G-invariant), so ~K\ is also Af(A^)-equivariant. It is easy to verify that im^ = 7-L K Hp . Let 
H K = imni © A, where A = H G . 

In [6] it has been shown that Problem [1] is equivalent to the following 

Problem 2. Let T G Cq(H,H). Do there exist A G A and u G im7Ti\{0} satisfying the 
equation (iri o T o z)(n, A) = 0? 

Define the operator A G C^^(im7Ti © A, im.7Ti) by A(u, A) = 7Ti(T(i(«, A))). It is easy to 
verify that the operator A is well defined. 

The following remark follows from the definition of 7Ti and the equality .4.(0, A) = for every 



Remark 2.1. If there exists a bifurcation point of solutions of the equation A(u, A) = 0, then 
the answer to Problem^ is affirmative. 

In view of remark 12.11 our aim is to study the bifurcations of solutions of the equation 
A(u, A) = 0. 

Throughout the rest of this section we will need the following assumptions: 



(3) L: H — > % is a linear, bounded, self-adjoint, G-equivariant Fredholm operator of in- 
dex 0, 

(4) Vr/ G Cq(T-L,T-L) is a completely continuous operator. 

From now on we put T = V$. Because the operator L is G-equivariant, L(1-L G ) C "H G , 
L(H K ) C 7i K ' . Since the operator L is self-adjoint, we obtain the following 



From the above we get 7Ti(L(i(u, A))) = 7ri(L(0, u, A)) = 7Ti(0, L 2 u, L 3 X) = L 2 u. Therefore 
A(u, A) = tti(V u $(m, A)) = L 2 u - 7Ti(V7?(i(«, A))). 

Lemma 2.2. For every A G A the operator A(- , A) G G^^-^iniTi"!, im^!) zs gradient. 
We refer the reader to [6J for the proof of the above lemma. 

From the above lemma it follows that the equation A(u, A) = has a variational and sym- 
metric structure. To study bifurcations of solutions of the equation can be used the degree 
for A^(A')-invariant strongly indefinite functionals V N(K)-deg(-, ■), which is an element of the 
Euler ring U(N(K)) of a compact Lie group N(K), see Appendix for the definitions and basic 
properties. 



A G A. 



(1) $G^,1), 

(2) $(u>) = l(Lw, w) - r)(w), 




L 



U 
L 2 
u 



H G U G 
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3. Elliptic system 

In this section we study the strongly indefinite functional associated with a system of elliptic 
equations. 

Consider the following system 

-Aw 1 — V Wl F(w 1 ,w 2 ) in Q 
Aw 2 = V W2 F{ Wl ,w 2 ) in n (3) 
^ = ** = on an, 

where 

(1) f2 is an open, bounded and G- invariant subset of an orthogonal G- representation W l , 
with a smooth boundary, 

(2) F E C7 2 (M 2 ,R), 

(3) \V 2 F(y)\ < a + b\y\ q , where a, b G R, g < ^2 for n > 3 and q < 00 for n = 2. 

Put in the previous section "H = iJ 1 (f2)©if 1 (f2). Since is an orthogonal G- representation 

with the action given by (g,u)(x) h-» u(g~ l x) for g E G, u E rr G f2, so is "H, where 

G acts on this space by ((?, (u,v))(x) i-» (u(g~ l x), v (g^x)) for g E G, u, v E H 1 ^), x G fi. 

Put L = ^1 ' k rev ^y we use the same notation for a matrix and the operator 

© ->■ © if^Q) induced by the matrix. 

Recall that a weak solution of the system is a function w E H such that 

V, ew y (LVw(i), Vv{x)) - (VF{w(x)),v(x))dx = 0, 

where (•, •) are the standard inner products in IR 2n and IR 2 . 
Put in the previous section 

= I J \Vw 1 (x)\ 2 - \Vw 2 {x)\ 2 dx - J F{w{x))dx (4) 
n n 

= \J |Vwi(V)| 2 - \Vw 2 {x)\ 2 + |u>i(x)| 2 - |w 2 (V)| 2 Gfe+ 
n 

/X I 
-|wi(x)| 2 - -|u> 2 (x)| 2 + F(w(x))dx = 

n 

1 /" 1 

— - I (V(Lw(x)), Vw(x)) + (Lw(x),w(x))dx — r)(w) = —{Lw, w)y, — r)(w), 
Z J z 



where 



/I 1 
-|wi(x)| 2 - -\w 2 (x)\ 2 + F(w(x))dx 
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and therefore 



(Vr](w),v) 



H 



(Lw(x),v(x)) + (VF(w(x)),v(x))dx. 



(5) 



Then $>(w) = \{Lw,w)t-i — v( w )i V$(w) = Lw — Vr/(w) and V77 is a completely continuous 
operator (and consequently compact). Moreover, a function w G "H is a weak solution of system 
03]) if and only if V$(w) = 0, that is w is a critical point of $. 

We study breaking of symmetries of critical orbits for the functional $. To do this, fix 
K G sub(G) and recall that we have defined an equivariant orthogonal projection 7Ti : ini7ri © 
A —7- ini7Ti, where im7Ti = H K T-L G and A = Hp . We have also defined the operator 
A G CL^(im7Ti © A,im7Ti) by A(u,X) = 7Ti(V$(z(tt, A))), that is 

*4(u, A) = 7n(V$(i(«, A))) = L 2 u - 7Ti(Vr](i(u, A))), 

where % is an embedding H K in {'H K )- L © % defined by = (0,x). From lemma I2T21 it 
follows that the operator A(-, A) G C^^^imvrx, im7Ti) is gradient for every A G A. 

Denote by cr(— A, $7) = {0 = ui < /12 < • . .} the set of eigenvalues of the elliptic equation 
on Q with the Neumann boundary condition and V_a (/■<*;) the eigenspace associated with 
Hh G cr(— A; O). We also use the following notation 

(1) = {0}, 

(2) % = V_ A (^ fc ) © V_ A (/i fc ) for fc£N, 

n 

(3) F=©% for nGN. 

fc=i 

Fix A G A = H G . We will calculate the degree Vjv(jf)- deg(y4.(-, A), B(im7Ti)), which is an 
element of the Euler ring U(N(K)). To do this we need to define an approximation scheme 
for the mapping A(-, A), see Appendix. Consider the sequence of A^(il~)-equivariant orthogonal 
projections T = {r n : H — > H : n G No} defined as follows 

(1) n>° = {o}, 

(2) W k = (Y-aM k e v_ A (u fc ) G ) © (V_ A (^) K e v_ A (/i fc ) G ) for fceN, 

n 

(3) = H' k for nGN, 

fe=i 

(4) r n is a projection such that imr n = "H /n , for nGN. 

Then T is an A^(i^)-equivariant approximation scheme on im7Ti = H K © H G . Moreover, 
ker L = U° and for every n G N U {0} it follows that r n o L = L o r n . Note that 7ri(?T) = H m . 
Consider the system: 

— Awi = awi + bw2 
A1U2 = bw\ + CW2 



duii 8w2 g 

du du 



111 
ill 
Oil 



n 
n 
on 



(6) 



and put A 



a 
b 



b 

c 



$(w) 



Then 

1 

2 



|V^i(x)| 2 - \Vw 2 (x)\ 2 - (A(w(x)),w(x))dx. 
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Note that from PJ and (jSJ) it follows that V$(w) = Lu> — Caw, where Ca is given by 

(Caw, v)h = J ((L + A)w(x),v(x))dx for w, v G H. 
u 



Lemma 3.1. For every w G Hk, v G H, (Caw, v)n = (j^(L + A)w, v) 



H ■ 



Proof. Note that L + A 



1 + a b 
b -1 + c 



and consider the formula: 



Then 



/ (V(L + A)w(x), Vu(x))dx 
n 

= / V((l + a)wi(x))Vfi(x) + V(6wi(x))Vui(a:)dx 
n 

+ / V(6«; 2 (z))Vu 2 (^) + V((-l + c)w 2 (x))Vv 2 (x)dx 
n 

= J (-A)((l + a)wi(x))vi(x) + (-A)(bw 1 (x))v 1 (x)dx 
n 

+ J(-A)(bw 2 (x))v 2 (x) + (-A)((-l + c)w 2 (x))v 2 (x)dx 
n 

= ilk /((l + a)u>i(x))fi(x) + + (6«; 2 (2;))u 2 (x) + ((-1 + c)w 2 (x))v 2 (x)dx 

a 

= Hk f ((L + A)w(x),v(x))dx. 



Therefore 



((L + A)w,v) n = (l + /i fc ) / ({L + A)w(x)v(x))dx = (1 + fj, h ){C A w, v) H . 



Hence {C A w,v) n = (j^(L + A)w,v) H . 



□ 



From the above lemma we obtain CA^Kk) C and therefore Ca'- Hk — > To de- 

scribe the restriction of A to subrepresentations of im7Ti, we first describe the restriction of 

1 - 



V$ to subrepresentations of H. Let Tk(A) 



b 



-1 



and a% k, a 2 k be the 



eigenvalues of the matrix Tk(A), fi^, / 2 ,fc the corresponding eigenvectors. Because the ma- 
trix Tk(A) is symmetric, a^k, a 2,k £ K.- Denote by ei, e 2 the standard base of R 2 . Then 
Hk = {(fi(x) ■ €1 + ip 2 (x) ■ e 2 : <fi G V_A(/Ufc)}- It is easy to check that 

{<fi(x) ■ ei + ip 2 (x) ■ e 2 : ipiE V_ A (/ifc)} = {<fi(x) ■ fi,k + <f2(x) • /2,/fc : V 2 * £ V_ A (/^)}- 



Hence we obtain (V$)|^ fc 



map. 



Id 

a 2 ,fcld 



where Id: Y^^i^k) V_a(a*&) is the identity 



Now we are able to describe the action of the restrictions of A(-, A) on the subrepresentations 
of im7Ti. Fix A G "H G and assume that dim Y~A(Hk) K Y-A(Hk) G > 0. Since A(u, A) = 
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7Ti(V$(i(u,A))), 

A\ K {u, A) = 7n(V$|« fc (z((«i, Ai), (u 2 , A 2 )))) = 7n(W| Wfc ((0, m, AO, (0, u 2 , A 2 ))) 

= 7ri(«i ifc ld(0, mi, Ai), (a 2 ,jfc ld(0, u 2 , A 2 ))) = (ai, fc ui, a 2 , k u 2 ), 
where (u, A) = ((u a , A a ), (u 2 , A 2 )) and (u h X,) G ((V_ A (^)^ © V_ A (^ fc ) G ) © V_ A (/i fc ) G ) for 



1,2. Therefore (>W( - > A)) 



ai,jfcld 
a 2 h Id 



(V-aW k 9V_aW g ). 

Define m°(Tfc(A)) = dimker Tk(A) and 



where Id: (V_ A (/i/ 6 V_ A (ft) fc ) ->• 



dim ker 








if dim(V_ A V_ A (/i fc ) G ) > o 
if dim(V_ A (^)^eV_ A (^ 



# \T . ( ... \G\ _ g 



Put 

oo „ oo 

= 5] m°(T fc (A)) and = £ m°(>V (-, A)). It is easy to see that: 

k=l k=l 

Lemma 3.2. V$ is an isomorphism if and only if i°(A) = 0. Fix A G A. „4(-,A) is an 
isomorphism if and only if i°(A) = 0. 

Naturally, if V$ is an isomorphism, so is A(-, A) for every A G A. 

Denote by m~(Tf.(A)) the Morse index of the matrix Tk(A). Note that m~ (Tk(A)) G {0, 1, 2} 
and for a sufficiently large k, m~(Tk(A)) = 1. Define the subspaces: 

v (A)= v_ A (/i fe ) x ev„ A (/i fc ) G , v 2 (A)= i A (ii/e?-AW G 

fe: m-(T k (A))=0 k: m-(T k (A))=2 

Theorem 3.3. Consider system satisfying i°(A) = and fix A G H G . Then 
V N{ K)-deg(A(-,\),B(imir 1 )) = V N{ K)-deg(-Id,B(V 2 (A)))*(V N{K )-fe 

Proof. From the definition of the degree, see formula ( fT5l) . for sufficiently large n the following 
equality holds 

V N ( K )-deg(A{-, A),-B(im7Ti)) 

= (V w deg(L 2 , ^ /0 ))) _1 * V N(K )- deg(A ]W n(; A), 

Note that from the product formula, see Appendix, and from the definition of the function L 2 
we obtain 

n 

Vjv(k)- deg(L 2 , B(H n H )) = V N{K) - deg(L 2 , 5(0 V_ A (^ fc ) x e V_ A (/i fe ) G )) 



fc=i 



V w - deg(- Id, B(V_ A (^i) A 6 y-^iT)) * V^- deg(- Id, 5(V_ A (/i 2 ) K e V_ A (/i 2 ) G )) 
* . . . * V w - deg(- Id, 5(V_ A (/i n ) K e V_ A (/i„) G )). 
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If m-{T k (A)) = 2, then 
V NiK) -deg(A lK (;\),B(H' k )) 

= V N(K) - deg((- Id, - Id), fl((V_ A e V_ A (/i fc ) G ) © (V_ A (/i fc ) x e V_ A (^) G ))) 

= V N{K) - deg(- Id, £(V_ A (^ fc )* e V_ A (/i fe ) G )) * Viv W - deg(- Id, B(V_ A e V_ A (/i fc ) G )) 

When m-(T fc (A)) = 1 

V N{K) - deg(A lWk (; A), = V N{K) - deg(- Id, 5(V_ A (/i fc ) x V_ A (^) G )). 

In the remaining case m - (T fe (A)) = we have Vn(k)~ deg(^4|-^(-, A), B(H' k )) = I, which com- 
pletes the proof. □ 

Consider the characteristic polynomial of T k (A) given by 

7 9 

a 



W fc (a fc ) = (1 - — a k )(-l - — a k ) 



1 + A*fc 1 + (l+/i fc ) 2 

It is easy to verify that 

Since the matrix T k {A) is symmetric, the polynomial has two real roots, denote them by 
®i,k, «2,fc- From Viete's formulae we get 

-(1 + fi k ) 2 + (a - c)(l + Hk) + ac - b 2 a + c 

oti, k a 2 , k = -— , ati k + a 2 ,fc 



(1 + ^) 2 ' ^ (1+/*)' 

Because the sign of the sum does not depend on k, the matrix T k (A) has the roots of the 
same sign if and only if (1 + /i fc ) 2 — (a — c)(l + /j, k ) — (ac — b 2 ) < 0. Solving the inequality 
(with respect to 1 + /j, k ) we obtain the discriminant 5 = (a — c) 2 + 4(ac — b 2 ) = (a + c) 2 — 4o 2 
and if 5 > 0, then ft = a ~ c ~^ - 1, /3 2 = a ~ c +^ - 1 are the roots of the polynomial 
(1 + fi k ) 2 - (a - c)(l + ;U fe ) - (ac - 6 2 ). If S < 0, then we put ft = ft = 0. Note that if 
(1 + fi k ) 2 — (a — c)(l + Aife) — (ac — 6 2 ) = for e cr(—S, Q), then cci^ = or cc2,fc = 0, so 
^ and if also dim(V_ A (/i fc ) x V_ A (/i fe ) G ) > 0, thenT°(A) ^ 0. 
Let P = er(— A, fi) fl (ft, ft) and note that for every fx k e P we have 

(1) If a + c < 0, then m _ (T fc (A)) = 2. 

(2) If a + c> 0, then m (T k (A)) = 0. 

Assume that i°(A) = 0. 

Theorem 3.4. Under the above notations and assumptions: 
(1) if a + c < 0, £/ien 

V^)-deg(^l(-, A), S(im7n)) = V JVW -deg(- Id, 5(0 V_ A (^) X e V„ A (/i) G )), 

MSP 
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(2) if a + c> 0, then 

Vj^-deg^A^BOmTT!)) = ( V N(K) -deg(- Id, 5(0 V_ A (//)* e V_ A (/i) G )) 

i/ the set P is empty, then Vn(k)~ deg(^4(-, A), S(im7Ti)) =1. 

Note that the condition i°(A) = is satisfied if and only if for every k G N 

-(1 + fi k ) 2 + (a - c)(l + /i fc ) + ac - &V 0. 

From now on we consider the following nonlinear system 

—Awi — V Wl F(wi,W2) in £1 
Aw 2 — V W2 F(w 1 , w 2 ) in Q 

^ = ^ = o on an, 

Recall that with this system is associated the functional $:%—>■ R given by 

= y I(|v Wl (x)| 2 - |Vw 2 (a;)| 2 ) - F(w(x))dx. 
n 

and V<&(w) = Lw — Vr)(w), where Vr] is an operator defined by 

(Vr)(w),v) n = J (Lw(x),v(x)) + (VF(w(x)),v(x))dx. 



(7) 



Moreover, V 2 $(w) = L — C^2f(w) for w G "H, where the operator C V 2f(w) : — > H is given by 
the equality 

(Cvzf(w)U, v) H = J {(L + V 2 F(w(x)))u(x),v(x))dx for u,v eH. 
n 

Denote Z = (VF) _1 (0) and let z G Z be a non-degenerate critical point of the functional $. 
Define 

(Cv2f(z)M, f )■« = J ({L + V 2 F(2;))-u(a;), t> (x))dx for u,v El-L. 



Denote V 2 F(^) 



a(z) 6(2) 
6(2) c(z) 



&(z) 



c(z) 



. Then T fc (V 2 F(^)) = 
a2,fe(^) be the (real) eigenvalues of the matrix Tk(V 2 F(z)). Then 



a ( z ) 



Kg) 



and let 



(V 2 $(^)W 



ai )fc (2;)Id 
a 2 ,fe(^)Id 



where Id: V^A^k) — > V_ a is the identical function. Note that for a sufficiently large k, 
m~(T k (V 2 F(z))) = l. 



10 PIOTR STEFANIAK 



Define A G H G by X(x) = z for every x G £1. Since the derivative of A with respect to u 
satisfies .44(0, A) = tx x o V 2 $(0, 0, A) oi, it follows that if dim(V_ A (^)^ V_ A (/ifc) G ) > 0, then 



a ltk (z)ld 
a 2 ,fc(^)Id 



'■k 

where Id: (V_ A (/i fe ) K V_ A (/i fc ) G ) -> (V_ A (/i fc ) x V_ A (/i fc ) G ). 
Define the subspaces 

v (v 2 f(z)) = v_ A (/i fc ) x e v_ A 0**)°, 

fc: m-(T fc (V 2 F(2)))=0 

V 2 (V 2 F(z)) = Y-aM k V_ A (/i fe ) G . 

fc: m-(r fc (V 2 F( 2 )))=2 

Theorem 3.5. Lei z E Z be such that i°(V 2 F(z)) = 0. Then there exists 70 such that for 
every < 7 < 70 

Vj^Kj-deg^-.A^B^imTri)) = V^-deg^O, A), S(imTTi)) 
= V N( K)-deg(-Id } B(V 2 (V 2 F(z)))*(V N(K )-deg(-Id,B(Vo(V 2 F(z)))) 



, -1 



Proof. We refer the reader to [13] for the proof of the first equality. The second equality follows 
from theorem 13.31 □ 

Put 5(z) = (a(z)-c(z)) 2 +A(a(z)c(z)-b(z) 2 ) = (a(z)+c(z)) 2 -Ab(z) 2 and if 5(z) > 0, then put 

(h(z) = a{z) - c{z l 2 (z) = <fhfW±^M _ L if < 0, then put = = 0. 
Define -P(^) = a(—A,Q) fl /^(z)). Similarly as before it can be shown that the matrix 

Tk(V 2 F(z)) has roots of the same sign if and only if G P(z). Moreover, for any G -P(z): 

(1) If a(z) + c(z) < 0, then m' (T k (V 2 F (z))) = 2. 

(2) If a(z) + c{z) > 0, then m-{T k {V 2 F{z))) = 0. 

Now we are able to give formulae of the degree of the functional associated with system (J7|). 
Assume that i°(y 2 F(z)) = 0. 

Theorem 3.6. Under the above notions and assumptions: 

(1) if a(z) + c(z) < 0, then for a sufficiently small 7 > 

V JV(Jf) -deg(^(-,A),S 7 (im7r 1 )) = V w -deg(-Id,S( ?-A(/i/ef- A (ft) G )), 

M eP( 2 ) 

(2) if a(z) + c(z) > 0, then for a sufficiently small 7 > 

Vjv(JO-deg(^(-,A),S 7 (ini7ri))= I V N{K) -deg(- Id, FJ( V_ A (/x fc ) x V_ A (/i fc ) G )) 

(3) if the set P(z) is empty, then for a sufficiently small 7 > 

Vjv(A-)-deg(^4(-,A),5 7 (im7ri)) = I. 
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The following theorem will be used to formulate a bifurcation theorem. The proof is similar 
in spirit to that of [TU]. We use the notation from this article. 

Lemma 3.7. Let the above assumptions hold and assume that the group G is connected. If Wi 
or W2 is a nontrivial G -representation, then 

V G - deg(- Id, S(Wi)) ^ V G - deg(- Id, B(W 2 ))-\ 

Proof. Recall that V G - deg(- Id, B{W t )) = Xg{S w >) for i = 1,2, see pLT], where S w * = 
D(Wi)/S(Wi) is a quotient of the closed unit ball and the unit sphere in Wj and xg de- 
notes the G-invariant Euler characteristic of S *. Suppose the assertion of the lemma is 
false, that is Xt(S Wi ) * Xt(S w ' 2 ) = I G E7(G), the unit in C/(G). Denote by T C G a 
maximal torus and notice that because Wj are G-representations, Wj can be also treated 
as T-representations. The natural homomorphism (p: T — > G induces a ring homomorphism 
(p*: U{T) -> U(G) such that (p*( X G(S Wi )) = Xr(S Wl )- Hence Xt(S Wi ) * Xt(S W2 ) = I G E7(T). 
From theorem 3.2 of [10] it follows that there exist k , ki, . . . , k r , k' Q , k[, . . . , k' r , G NU {0}, 
K mi ,. . . , K mr , K m ' x , . . . , Km' r , G sub(T), x, x', y, y' G U(T) such that 

(1) dim K m = dimi^ m ' = dimT — 1 for every i, 

(2) x = £k l -x T (T/K m J, 

8=1 

(3) x' = Y,k' i - X T{T/K m ,), 

(4) y= 1 n ( K ) ■ Xt(T/K+), where n(K) G Z, 

(Jf)6{(X:)6sub[T]:dimX:<dimT-l} 

(5) y' = £ n'(tf) ■ xt(T/K+), where n'(iT) G Z, 

(X)e{(/C)esub[T]:dim/C<dimT-l} 

(6) Xt(5 Wi ) = (-l) fe °I + (-l) fe( >x + y, 

(7) Xt(^ W2 ) = (-l) fc oI + (-1)*^ + ?/. 

From the assumptions (since G is connected) it follows that x ^ I or x' ^ I. It is easy to see 
that 

((-l) fco I + (-l) ko X + ?/)((-l) fc 0l + (-l) fc X ' + y>) 
— (_l)fco+fc ( ',]J _|_ (—l) k °+ k Q X + ( — l) feo+fc OX / 

+(-l)*oj/ + (-l) fc y + (-l) fco+fc o xx ' + (-l) fc °xy' + (-l) k '°x'y + yy'. 
Note that if k Q + k' Q is even, then (— l) fc o+ fc o = 1 an d 

(_l)*o+*&I + (_l)*o+*ta + ^_ 1 )fc +^ x / = J + x + x / _^ J_ 

If k + k' is odd, then (-l) fc °+ fe o = -1 and 

(_1)*d+*&i + (_i)*o+*oa; + (-ijfco+fcj^ = _i _ a; _ x > ^ 1 

From the properties of the multiplication in U(T), see [TU], it is easy to verify that 

(-l)S + (-1) V + xx' + xy' + x'y + yy' = £ n"(#) • X t(T/K + ), 

(K)e{(/C)esub[T]:dimK:<dimT-l} 

where n"(^) G Z. Therefore Xt(S Wi ) * Yt(S W2 ) = ((-l) fc °I + x + y)((-l) k '°I + x' + y') ^ I. 
This contradiction completes the proof. □ 
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4. The main results 

In this section we apply the bifurcation theory to formulate conditions implying breaking of 
symmetries. That, is we formulate theorems to answer the problem: does there exist a function 
/ = h) £ H G such that the system 

-Aw 1 = V Wl F(w 1 ,w 2 ) + fi in Q 

Aw 2 = V W2 F(w u w 2 ) + f 2 in VI (8) 

au av 

has a weak solution w G T-L K \H G 7 

Recall that $: H 1 ^) x 'H 1 (r2) — > R is the functional associated with the above system and 

(1) Q is an open, bounded and G- invariant subset of an orthogonal G- representation MJ 1 , 
with a smooth boundary, 

(2) F G C 2 (M 2 ,M), 

(3) \V 2 F(y)\ <a + b\y\ q , where a, b G R, g < ^ for n > 3 and q < oo for n = 2. 

Assume that the set 2 = (V-F) _1 (0) is finite and fkzfZ. In the previous sections we have 

a(z) b(z) 



introduced the following notation: V F{z) 



b(z) c(z) 



, S(z) = (a(z) + c(z)f - Ab(zf 



and if 8{z) > 0, then 1 (z) = ^hf^zVgg _ 1; ^ = ^)~< Z )+^W) _ L ff ^ < ^ ^ 
= /3 2 (z) = 0. We have also put P(z) = a(-A,fl) n 
Denote 

mi = mm{(3i(z) : z G Z}, Mi = max{ / 9 1 (2;) : z G Z}, 
W2 = min{f3 2 (z) : z G Z}, M 2 = m&x{j3 2 (z) : z G Z}. 
Assume that P{z) ^ and 7 > is a sufficiently small number. By theorem 13.61 we obtain 
V7v(ic)-deg(^4(-, A),.B 7 (im7ri)) = 

V W -deg(-Id,B( © (V_ A (#eV_ A (/i) G ))),whena(z) + c( 2 )<0 

V W - deg(- Id, B{ © (V_ A (/x) x e V_ A (/i) G ))) ] , when a{z) + 0(2) > 0. 
M eP( 2 ) / 

(9) 

If P(z) = 0, then V N{K) -deg(A(; A), B^imnJ) = I G U(N(K)). 

Recall that for every A G A, .4.(0, A) = 0. In the following theorems we formulate conditions 
implying a global bifurcation of solutions of the equation A(u, A) = 0. To do this we use 
coefficients of the matrix V 2 F(z). We use the notation: if zi, z 2 G Z, then Ax, A2 G "H G are 
defined by Ai(x) = zi and X 2 (x) = z 2 for every x G fl. 

Theorem 4.1. Let the above assumptions hold. Assume that there exist zi, z 2 G Z such that 

(1) P{zi) 7^ and © V_a(aO ^ s a nontrivial N (K) -representation or P{z 2 ) 7^ and 

© V_a(a*) is a nontrivial N(K) -representation, 

(2) a(zi) + c(zi) < < a(z 2 ) + c(z 2 ), 
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(3) i°(V 2 F( Zl )) = i°(V 2 F(z 2 )) = 0, 

(4) N(K) is a connect group. 

Then there exists a global bifurcation point of solutions of the equation A(u, A) = 0. 
Proof. From formula fl9]) it follows that for sufficiently small 71, 72 > : 

V N(K) - deg{A{; Ax), £ 71 (im m)) = V N{K) - deg(- Id, B{ (V_ A (/*)* 6 V_ A (/i) G ))) 

AteP(*i) 

and 

V2 V( jo-deg(^(.,A 2 ),ST Sl (im7r 1 )) = V w -deg(-Id,B( (tif/elAfii) 6 )))- 1 

From lemma 13.71 we obtain that 

V N (K)-deg(A{-, Ai),S 7l (im7ri)) 7^ Viv(/f)-deg(,4(-, A 2 ), /3 72 (im7Ti)). 

Therefore the theorem follows from theorem 16.41 □ 

Also, if between rri\ and M\ exists fi G cr(— A, Q), then there can appear breaking of symme- 
tries. 

Theorem 4.2. Suppose that for every z G Z either a(z) + c(z) > or a(z) + c(z) < 0. 
Let z\, Z2 E Z be such that mi = Pi(zi), M\ = Pifa) and assume that i°(V 2 F(zi)) = 
i°(V 2 F(z2)) = 0. Suppose that one of the following conditions is satisfied: 

(1) There exists fa G cr(— A,Q) such that m\ < fa < Mi, where V-aO^i) 6 V^a(^) G 

a nontrivial N(K) -representation and for every fa; G cr(— A, fi) if 1112 < fa; < M2, then 
V-A(/ij)^0V_A(ft') G is a trivial, ev en- dimensional N{K) -representation. 

(2) There exist fa, fa G a(— A, fl) such that mi < fa < Mi and fai < 7772 < M 2 < fa+i, 
where Y-A(fa) K Q^-A(fa) G is a nontrivial N(K) -representation. 

Then there exists a global bifurcation point of solutions of the equation A(u, A) = 0. 

Proof. It easy to see that both the assumptions imply that for sufficiently small 71, 72 > 

Vjv(ftr)-deg(^4(-, Ai),-B 7l (im7Ti)) ^ V N{K) - deg(^4(-, A 2 ), 5 72 (im7Ti)). 

Therefore the thesis follows from theorem 16.41 □ 

Analogously as the previous theorem, if between 777,2 and M 2 exists /i G a(— A,Q) we can 
formulate and prove conditions implying breaking of symmetries. 

Theorem 4.3. Suppose that for every z G Z either a(z) + c(z) > or a(z) + c(z) < 0. 
Let zi, Z2 G Z be such that m 2 = ^2(21), M 2 = ^2^2) and assume that i°(V 2 F(zi)) = 
i°(V 2 F(z2)) = 0. Suppose that one of the following conditions is satisfied: 

(1) There exists fa. G cr(— A,Vt) such that m 2 < faj < M 2 , where Y-&(fa) K V_a(/-^') G is 
a nontrivial N(K) -representation and for every fa G cr(— A,Q) if mi < fa < Mi, then 
Y-A{fa) K ^-A{fa) G is a trivial, even- dimensional N(K) -representation. 

(2) There exist fa, fa G er(— A,Q) such that fa < m\ < Mi < fa + i and m 2 < fa: < M 2 , 
where V. A (fa) K V_ A (/i;) G is a nontrivial N(K) -representation. 
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Then there exists a global bifurcation point of solutions of the equation A(u, A) = 0. 

Recall that if there exists a global bifurcation point of solutions of the equation A(u, A) = 0, 
then there exists A G A such that the connected component C(A ) in the closure of the set 
J\f = {(u,X) G (iniTTi \ {0}) x A : A(u, A) = 0} satisfies C(A ) ^ {(0,A )} and either C(A ) 
is not bounded or C(Ao) fl (0 x (A \ {Ao}) ^ 0. In particular, we obtain a connected set of 
solutions of our main problem. 

Moreover, for every (u, A) G C(A ) \ ({0} x A), the equality A(u, A) = implies that 
7Ti(V$(i(u,A))) = 0, so V$(i(u,A)) G % G and therefore there exists / = (/i,/ 2 ) G % G such 
that the system 

-Aw 1 = V wl F(w 1 ,w 2 ) + fi in fi 

Aw 2 = V W2 F(wi,w 2 ) + f 2 in (10) 

ov ov 

has a solution in H K \H G . That is, i(C(\ ) \ ({0} x A)) is a set of solutions of the main problem. 
Moreover, we have obtained that the whole connected set i(C(A )) is mapped by V<3> into a 
connected set in "H G . 

5. Examples 

In this section we show an application of the bifurcation theorems. Consider the following 

system 

' -Awi=V Wl F(w u w 2 ) in B n 

Aw 2 = V W2 F(wi,w 2 ) in B n (11) 
^ = ^ = on dS n -\ 
where 

(1) B n is an open unit ball in an orthogonal SO(n)-representation W l , where SO(n) is a 
special orthogonal group, 

(2) F G C 2 (M 2 ,M), 

(3) \V 2 F{y)\ < a + b\y\ q , where a, b G R, q < f° r n — 3 an d g < oo for n = 2. 

Denote 2 = (V-F) _1 (0) and define the function (p: R x Z — > K. by the formula </?(x, z) = 
-a; 2 + (a(z) - c(z))x + a(*)c(z) - 6(^) 2 . Put K = {1 + G <r(-A, 5 n )}. Recall that the set 
er(— A, _B n ) is discrete and therefore so is the set /C. 

Theorem 5.1. Suppose that there exist zi, z 2 G Z such that 

(1) Zi y\0) nlC = (l} fort = 1,2, 

(2) P(zi) 7^ and V_a(/-0 ^ o nontrivial N(K) -representation or P(z 2 ) ^ and 

//6P(2l) 

V_a(aO ^ s a nontrivial N(K) -representation, 

(3) a{zi) + c(zi) < < a(z 2 ) + c(z 2 ). 

Then there exists a connected set of points in H K \H G such that for each point from this set 
there exists f G % G such that these points are solutions of the system 

-Awi = V wl F(wi,w 2 ) + fi in il 
Aw 2 = V W2 F(wi,w 2 ) + f 2 in fi (12) 

ov ov 
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Proof. The first assumption implies that i°(V 2 F(zi)) = i°(V 2 F(z2)) = 0, hence i°(V 2 F(zi)) = 
2°(V 2 -F(z 2 )) = 0. Theorem 14. II completes the proof. □ 

Elements of the set <j{— A,B n ) are well known, especially if n = 2,3, see [IB]. So are 
representations of SO(n), for n = 2, 3, and the degree for SO(2)-invariant functionals. Therefore 
the assumptions of theorem 15.11 are easy to verify. 

Remark 5.2. (1) If G = SO(n) and K = {e} (the trivial subgroup), then the normalizer 
of K is equal to SO(ra) and H K = H. Therefore using our method we can study existing 
of non-radial solutions such that V$(u>) G "H SO(n ). 

(2) If G = SO (3) and K = Z m (a cyclic group), then the Weyl group of K is equal to 0(2) 
and the space 7i K is an 0(2) -representation and therefore an SO (2) -representation. 
Using the degree for SO (2) -equivariant maps we can study the problem: does there exist 
w G n Zm \n so ^ such that V$H G "H so(3) ? 

(3) In [BJ has been considered problem of breaking symmetries for elliptic equation using 
the homotopy index. The author has obtained a sequence of solution of the bifurcation 
problem. We emphasize that using the degree for gradient G-equivariant maps, we have 
obtained a connected set of solutions. 

(4) In this article we have considered a simple example of the method, but the same could 
be used for more complicated equations. 

6. Appendix 

To make this article self-contained we recall the definitions and properties of the Euler ring 
for a compact Lie group and the degree for G-invariant strongly indefinite functionals. 

Denote by J-*(G) the class of pointed G-CW-complexes, see [H] for the definition, and by 
[X] the G- homotopy class of a pointed G-CW-complex X. Let F be the free abelian group 
generated by the pointed G-homotopy types of finite G-CW-complexes and N the subgroup of 
F generated by all elements [A] — [X] + [X/A] for pointed G-CW-subcomplexes A of a pointed 
G-CW-complex X. 

Definition 6.1. Put U(G) = F/N and let Xg{X) be the class of [X] in 17(G). The element 
Xg{X) is said to be the G-equivariant Euler characteristic of a pointed G-CW-complex X. 

For X, Y G F*{G) let [XVY] denote a G-homotopy type of the wedge IV7gJ,(G). Since 
[X] -[XVY] + {{X V Y)/X\ = [X] - [X V Y] + [Y] G N 

Xg(X) + Xg (Y) = xg(XVY). (13) 

For X, Y G J**(G) let X A Y = X x jX V Y, The assignment (X, Y) 4lAY induces a 
product U(G) x U(G) -> U(G) given by 

Xg(X)*xg(Y) = X g(XAY). (14) 

If X is a G-CW-complex without a base point, then by X + we denote a pointed G-CW- 
complex X U {*} and consequently we put xg(X) = xg(X + ). 

Lemma 6.1. (U(G),+, *) with an additive and multiplicative structures given by f jl~3|) . f JT4|) , 

respectively, is a commutative ring with unit I = xg{G/G + ). 
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We call (U(G), +, ★) the Euler ring of G. 

Denote by sub[G] the set of conjugacy classes of subgroups of a group G. 
Lemma 6.2. (U(G),+) is a free abelian group with basis xg{G/K + ), (K) G sub[G]. 

See [8j [9] for the complete definition and more properties of the Euler ring. 

An element of the Euler ring is the degree for G-invariant strongly indefinite functionals, we 
recall the definition and properties. Let ("H, (•, •)) be an infinite-dimensional, separable Hilbert 
space which is an orthogonal G-representation. Denote by T = {r n : H — > H : n G N U {0}} a 
sequence of G-equivariant orthogonal projections. 

Definition 6.2. A set T is said to be a G-equivariant approximation scheme on H if 

(1) for every n G N U {0}, H n is a finite subrepresentation of the representation 

(2) H n+l =U n ® U n+ i and n n ±n n+1 , 

(3) for every u G H lim r n {u) = u. 

n— yoo 

Assume that 

(al): Q C % is an open, bounded and G-invariant subset, 

(a2): L: H —> H is a linear, bounded, self-adjoint, G-equivariant Fredholm operator 
satisfying the following assumptions: 

(a) kerL = ^°, 

(b) 7r n o L = L o 7c n , for all ?2 G N U {0}, 

(a3): V77: Q — > H is a continuous, G-equivariant, compact operator, 
(a4): $ G Cq(Q,M) satisfies the following assumptions: 

(a) V$(w) = Lu- Vr](u), 

(b) d((v$)- 1 (o))n^ = 0. 

Under the above assumptions define the degree for G-invariant strongly indefinite functionals 

by 

V G - deg(L - V77, Q) = (V G - deg(L, B{H n Q H°))Y l * V G - deg(L - 7i n V V , Q e n H n ), (15) 
where e > is sufficiently small and n G N is sufficiently large, see [13] for details. 

Theorem 6.3. The degree has the following properties: 

(1) (a) z/V G -deg(V$,fi) ^ G U(G), then (V^)- 1 ^) Hft ^ 0, 

(b) i/n = OiUfi2 and Qi, Q2 are open, disjoint and G-invariant sets, then 

V G - deg(V$, to) = V G - deg(V$, fli) + V G - deg(V$, fi 2 ), 

(c) if Qi C Q is an open and G-invariant set and (V$) _1 (0) fl C fli, i/ien 

V G -deg(V<l»,fi) = V G -deg(V$,f2i), 

(dj if £ £1 and $ G G G (fi,M) stjc/z. tfitrf V$(0) = and V 2 $(0): U ^ U is a G- 
equivariant self-adjoint isomorphism then there is 70 > such that for every 7 < 70 
we have 

V G -deg(V$,B 7 (ft)) = VG-deg(V$ 2 (0),5(H)). 
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(2) Fix $ G C X G {U x [0,1], E) suc/i tfm* (V u $) _1 (0) D x [0,1]) = and V u ${u,t) = 
Lu — V u r](u,t), where V u i]: Q x [0, 1] — > T-L is G-equivariant and compact. Then 

V G -deg(V M $(-, 0), Q) = V G -deg(V u $(-, 1), fi). 

Let Q\ C 7^i, SI2 C H.2 he open, bounded and G-invariant subsets of G -representations 
%i, %2- Assume that the Junctionals G C G (Hj,M), i = 1,2 are 0/ £/ie /orm 
= h(LiU,u) + T)i(u) and satisfy the assumptions (al)-(aJf.). Define a functional 
$ G C G (%i©% 2 ,R) fry $(ui,u 2 ) = + $("2) and set Vt = Sl x x fi 2 - ^/ien 

V G - deg(V$, fl) = Vg- deg(V$ 1 , fix) * V G - deg(V$ 2 , fi 2 ). 

Theorem 6.4. Fzx $ G C G (% x A,R) such that V u $(tt, A) = Lu - V u rj(u,\), where the 
mapping V u f]: O x A H is G-equivariant and compact. Suppose that V M $(0, A) = for 
every A G A. If there exist 71, 72 > such that 

V G -deg(V u <l>(-,A 1 ), J B 7l (H)) ^ V G -deg(V u $(-,A 2 ), J B 7 (H)), 

i/ien at every path joining (0, Ai) and (0, A 2 ) exists a global bifurcation point of solutions of the 
equation V u $(u, A) = 0. 

See [13] for properties of the degree and [Tl] [21] for the definition of the degree for gradient 
G-equivariant maps. For the general theory of the equivariant degree we refer the reader to pQ , 

m. 
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